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ABSTRACT 

\ 

Given  a  random  field  belonging  to  some  specific  class,  and  given  a  data  sample  gen¬ 
erated  by  the  random  field,  consider.^the  problem  of  finding  a  field  of  the  given  class 
that  approximates  the  field  that  generated  the  sample.  This  paper  derives  a  solution  to 
this  problem  for  the  simple  case  of  a  field  consisting  of  independent  random  variables. 
Subsequent  papers  will  treat  other  types  of  fields,  e.g.,  having  Markov  dependencies. 
Numerical  examples  are  given,  showing  that  good  approximations  can  be  obtained  based 
on  relatively  small  sample  sizes.  In  particular,  this  approach  can  be  used  to  find  random 
field  models  that  generate  given  samples  of  image  texture,  and  so  can  be  applied  to  tex¬ 
ture  classification  or  segmentation. 
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1.  Introduction 


1.1.  The  purpose  of  this  research 

In  what  follows  we  will  consider  the  following 

Problem: 

Given  a  sample,  determine  the  random  field  that  generated  it. 

At  first  glance,  this  problem  seems  to  be  without  solution,  because  of  the  lack  of 
sufficient  data.  In  order  to  make  the  problem  reasonable,  it  is  necessary  to 
assume  that  the  field  is  not  arbitrary  but  belongs  to  some  specific  class,  e.g., 

1)  is  composed  of  independent  random  variables 

2)  is  first  order  Markov  (e.g.,  in  two  dimensions,  it  is  a  Kanal  mesh  [l]  [2]) 

3)  is  n-th  order  Markov 

4)  is  weakly  (second  order)  stationary 

5)  is  strongly  stationary 
and  so  on. 

Making  one  of  these  assumptions  means  that  in  reality  we  are  not  consider¬ 
ing  the  problem  of  finding  the  field  that  generated  the  given  sample,  but  some 
other  field  that  belongs  to  the  given  class  and  approximates  the  field  that  gen¬ 
erated  the  given  sample.  In  this  paper  we  will  not  be  interested  in  the  problem 
of  evaluating  how  good  this  approximation  is,  because  this  aspect  is  treated  in 
the  author’s  papers  [8]  [9j. 


In  this  series  of  papers  we  will  study  the  problem  in  the  case  where  the  field 
is  assumed  to  be  strongly  stationary,  with  some  additional  restrictions. 

1.2.  Direction  of  the  research 

The  first  part  of  our  research,  presented  in  this  paper,  is  concerned  with  the 
simplest  case,  where  we  have  a  stationary  field  made  up  of  independent  random 
variables;  obviously,  we  may  suppose  that  the  field  is  one-dimensional. 

The  next  stage  of  the  research  will  consider  the  case  of  one-dimensional  sim¬ 
ple  homogeneous  Markov  chains,  followed  by  one-dimensional  Markov  homogene¬ 
ous  chains  of  higher  order.  Subsequent  stages  will  study  two-  (or  higher-)  dimen¬ 
sional  Markov  random  fields  (Kanal  meshes),  simple  or  of  higher  orders. 

1.3.  Digitization 

In  order  to  be  able  to  deal  with  digitized  data  and  at  the  same  time  to 
reduce  the  complexity  of  the  problem,  we  will  consider  only  random  fields  with  a 
finite  set  of  possible  outcomes  at  each  point.  In  order  to  extend  these  results  to 
the  continuous  case,  we  would  have  to  consider  some  process  of  approximation, 
such  as  that  used  by  the  author  [3]— [7]. 

2.  The  direct  theorem 

2.1.  Generalities 

Let  us  consider  a  sequence  of  independent  trials  with  possible  outcomes 
A,  (l  <  i  <n)  and  corresponding  probabilities  p,  >  0  (1  <  «  <  n)  adding  up  to 
1.  Each  possible  result  of  a  series  of  s  consecutive  trials  can  be  written  as  a 


2 


sequence 


Ct  —  (Aki,  A^,  .  .  .  ,  Ak)  (2.1) 

where  each  kr  (1  <  r  <  s)  can  take  any  value  » (1  <  t  <  n).  Because  of  sta- 
tionarity,  the  probability  of  occurrence  of  the  sequence  Cs  does  not  depend  on  the 
moment  when  the  trials  begin;  taking  into  consideration  the  independence  of  the 
trials,  this  probability  can  be  written  as 


p(c,)  =  n  put) 

r=l 


Let  us  denote  by  m,-  (1  <  i  <  n)  the  number  of  times  the  outcome  A,  appears  in 
the  sequence  Ca,  so  that 

E  mi  =  *  •  (2.3) 

i=l 

The  equality  (2.2)  can  be  written 


^  (<?,)  =  n  p?'  (2.4) 

1=1 

In  what  follows  we  denote  by 

H  —  E  P>  loS  “  (2.5) 

i=i  ”  i 

the  entropy  of  the  random  field  characterized  by  the  probabilities  p,  (l<  t  <  n). 


P  =  E  i°s  — 


(2.6) 


Obviously 


0  <  p  <  oo  (2.7) 

2.2.  The  theorem 

Let  us  denote  by  the  class  of  all  sequences  Ca.  For  given  6  >  0,  s  >  0  we 
denote  by  T  j  4  the  set  of  all  sequences  Ca  6  T4  such  that 

|  m,  -  sp,|  <  s6  (2.8) 

for  all  i  (1  <  i  <  n),  and  by  r"t  its  complement  with  respect  to  r4. 

Definition.  Sequences  Cg  G  r^4  will  be  called  (6,$)-standard  sequences  or  simple 
standard  sequences. 


Let  us  consider  the  equation 


i-f 

2: r  Jn 


\/27r 


«  — 

e 


2  dx 


and  let  us  denote  by  u  (e)  its  solution. 


Definition.  Given  e  >  0,  6  >  0,  s  >  n,  condition  A  holds  if 


4  iS2  e  s  >  n 


and  condition  B  holds  if 


4  (5 2  s  >  «2  (e) 


(2.9) 


(2.10) 


(2.11) 


Let  us  denote  by  N  (  )  the  cardinality  of  a  set. 


Theorem  1. 


Let  us  suppose  that  at  least  one  of  the  conditions  A,  B  holds.  Then 
(a)  If  Ct  is  a  (6,s)-standard  sequence,  it  follows  that 


1 


— -  log 

«  8  P{C.) 


-n\<6p 


(b ) P{rt',)  >  l-c 

(c)  lim  -log  N(Ti,)  =  H 

I  -*  00  S 
6  -*  0 

Remark  1. 

The  relation  (2.12)  is  equivalent  to 


2-»  [H  +  S/>)  <  p  j  2-‘  (H-ip) 


(2.12) 

(2.13) 

(2.14) 


(2.15) 


i.e.  to 


P{Ct )  =  2~sH+t$p0  ,  \0\  <  1 


(2.16) 


Remark  2. 


The  relation  (2.13)  is  equivalent  to 


pm  <  € 


(2.17) 


Remark  3. 


From  (2.14)  it  follows  that 

*07.,) 


lim  =  0 

,-.00  N{T,) 


."“oo  N( rj 


=  1 


(2.18) 


Indeed,  from  (2.14)  we  obtain  the  relation 


log  N  (17  ,)  = 

i.e., 

Taking  into  consideration  that 

N  (T  J  =  * 

and  because 


' -Iff*  0(1) 

(2.19) 

,,{H+  0(1)) 

(2.20) 

_  2*  log  n 

(2.21) 

if  follows  that 


II  <  log  n  , 


_  2_4(|og  n-  H  +  o(l)) 

iV(  rj 


(2.22) 


(2.23) 


which  is  equivalent  to  the  first  equality  in  (2.18),  and 

n(i X)  N(rtyN(riit)  ,  N(r(4)  ,  _ 
-iv(rr  =  — }v(r^ - 1  -  W  =  1  +  0(1) 


(2.24) 


which  is  equivalent  to  the  second  equality  in  (2.18). 

Remark  4. 


Our  Theorem  1  is  closely  related  to  some  results  which  go  back  to  Shannon 
[10]  and  received  a  mathematically  acceptable  form  from  Khinchine  [2]. 

Our  Theorem  1(a),  (b)  refers  to  independent  random  variables,  while  that  in 
[2]  refers  to  ergodic  simple  Markov  chains,  but  our  result  is  not  a  particular  case 
of  that  in  [2].  Indeed,  the  results  in  [2]  are  existence  theorems,  considering  that  <5, 
e  can  be  taken  as  small  and  s  as  large  as  desired,  while  our  results  give  effective 
relations  between  <5,  e,  s  in  order  that  the  results  hold. 


0 


Our  Theorem  1(c)  refers  to  the  set  r/4  of  all  standard  sequences  Ca,  while 
the  result  in  ([2],  Th.  3)  refers  to  another  set  of  sequences  Ca;  our  result  contains 


a  limit  for  6  — *  0,  s  — ►  oo,  while  the  result  in  ([2],  Th.  3)  contains  a  limit  for 
s  —*  oo. 

2.3.  Proof 

(a)  Let  us  consider  a  sequence  Ca  6  T/a.  From  (2.8)  it  follows  that 

m,-  =  sPi  +  sMi  \6i\  <1  (1  <  i  <  n)  (2.25) 


From  (2.4)  there  follows  the  relation 


log  P(Ca)=  2  m,  log  Pi 

i=i 

and  taking  into  consideration  (2.25),  there  follows  the  equality 
log  P  (CJ  =  £  (sp,  +  s6&i)  log  Pi 

i=i 

n  n 

=  s  £  Pi  lo6  Pi  +  s6  •  £  0,  log  Pi 

i=i  >=i 


(2.26) 


(2.27) 


which  can  also  be  written  as 


log 


1 

P(C,) 


=  sH  +  s6  ■  £  $i  log 

i=i 


_1_ 

P. 


(2.28) 


From  (2.28)  we  obtain  the  result  (a): 


P(Ct) 


-H\ 


<6  -  2  K  I  log  —  <  <5  •  £  log  — 
i=l  Pi  i=l  Pi 


Sp  (2.29) 


(b)  Instead  of  proving  inequality  (2.13)  we  will  prove  (2.17).  In  order  that  a 
sequence  Ca  6  T4  belong  to  r"4,  it  is  necessary  that  for  at  least  some  value 
of  t  (1  <  i  <  n)  the  inequality  (2.8)  does  not  hold,  i.e., 


r"s  =  U  {lmi  -  »P.I  >  «*} 


(2.30) 


so  that 


P{r6,s)  =  U  {in*,-  -  *P,I  >  5<5}}  ^  E  p{K~  SP.I  >  s<5[  (2.31) 

(bl)  Let  us  assume  that  condition  A  holds.  It  is  known  from  the  elements  of  the 
Theory  of  probability  that 


’{  K  -  sp,| 


>  s6  < 


Pi  (!— P») 


s^2 


(2.32) 


But  for  0  <  x  <  1,  we  have  the  inequalities 


0  <  x  (1  -  x)  <  — 
_  v  ;  -  4 


(2.33) 


where  the  maximum  value  is  reached  for  x  —  — ,  so  that  from  (2.32)  it  fol¬ 


lows  that 


p|  i mr  sp,|  >  s<5  J  <  -^7  (1  <  t  <  n) 


(2.3-1) 


Consequently,  from  (2.31)  there  follows  the  inequality 


pin.)  < 


n 


4s6‘ 


(2.35) 


and  because  of  (2.10),  it  follows  that  (2.17)  holds. 


(b2)  Let  us  assume  that  condition  B  holds.  From  the  Central  limit  theorem  in 


the  Moivre-Laplace  form,  it  is  known  that 


(2.36) 


so  that 


b\J  pAi'-\) 


2  ,  (2-37) 

e  ax  ,  (1  <  i  <  n) 


In  order  to  obtain  the  relation  (2.17)  it  is  sufficient  to  take 


1  - 


^  Jo 


L 


6\/  p,  (i  -  p.) 


e  *  dx  <  —  (1  <  t <  n) 

n 


(2.38) 


i.e., 


.  f‘V«M  -i-  i  ,  , 

wi  ' 2  ix>~.  'I‘- -i  (I 


(2.39) 


which  is  equivalent  to  the  inequality 


s 

p,  (i -p^  >  « (0  (i  <  *  <  «) 


(2.40) 


Because  of  (2.33),  we  have  the  inequality 


s 

p,  (1  -p()  >  2 <5/?  (1  <  /  <  n) 


(2.41) 


9 


so  that  in  order  to  satisfy  (2.40)  it  is  sufficient  to  take  in  consideration  Con¬ 
dition  B  (2.11),  i.e. 

2S\/s  >  u  (e)  .  (2.42) 

(c)  If  Cs  €  r$  s,  then  (2.15)  holds,  so  that 

jv(r(,)2 -(»-*<■)  <  £  P(c,)  =  P(rj,)  <  i  (2.43) 

where  the  summation  is  for  all  Ca  €  r/a.  From  (2.43)  there  follows  the  rela¬ 
tion 

-  log  N(rjt)<H+6p  (2.44) 

In  a  similar  way,  from  (2.13),  (2.15)  there  follow  the  relations 

1  -  e  <  P(rf't)  =  £  P(C4)  <  N(rU  2“4(//'^  (2.45) 

where  the  summation  is  also  for  all  Ca  ETfa.  From  (2.45)  we  obtain  the 
relation 

H-Sp  <-  log  N{T'6a)  +  -  log  -J—  (2.46) 

s  s  1  -  t 

From  (2.44),  (2.46)  it  follows  that 

H  -  bp  -  —  log  — - —  <  —  log  N(Tfa)  <  H  +  bp  (2.47) 

s  1  -  e  s 

For  e  given,  arbitrary,  6  as  small  as  we  want,  and  s  as  large  as  we  want, 
because  of  (2.7)  it  follows  that  (2.14)  holds. 


3.  The  inverse  theorem 


3.1.  Generalities 

Let  6  >  0,  e  >  0,  s  >  1,  and  let  C%  be  an  arbitrary  specific  sequence, 
belonging  to  1%.  Let  us  assume  that  one  of  the  conditions  A  or  B  holds. 

In  what  follows  we  assume  that  is  generated  by  a  sequence  of  indepen¬ 
dent  trials,  with  possible  outcomes  A,  (1  <  i  <  n)  with  unknown  probabilities 
P«  (1  <  i  <  n),  and  we  will  try  to  determine  some  intervals  in  which  these  proba¬ 
bilities  can  take  values.  Let  us  denote 

m?  =  m,  (C®)  (1  <  »  <  n)  (3.1) 

and  by  W{5)  the  confidence  of  statement  S. 

3.2.  The  theorem 

Because  we  have  proved  that 

P(rj  j  >  i  ~  e,  p  (r", )  <  e  (3.2) 

it  follows  that  with  confidence  larger  than  1  -  e,  €  r/4,  i.e., 

| m°  -  sp,l  <  6s  ,  (1  <  i  <  n)|  >  1  -  e  (3.3) 

i.e., 

f  m<t  1  ro  a\ 

W\  —  -  6  <  Pi  <  —  +  6  ,  (1  <  *  <  n)  |  >  1  -  €  (3-4) 

Is  s  ' 

Let  Ln  be  the  Banach  space  of  all  vectors 


with  g,  real  numbers  of  any  sign,  with  norm 


\\q\\  =  sup  J|?,|;  1  <  »  <  nj 

Let  IIn  be  the  totality  of  probability  measures 

P  —  (P„  •  •  •  ,  Pn) 

with  Pi  >  0  (1  <  »  <  n),  and 

t  P.  ^1 

i=i 

This  is  a  metric  space  with  distance 

||p  -  p'l!  =  sup  |  |p,  -  pI |;  l  <  *  <  »J 

where  p,  p'  E  nn  p  -  p'  €  Ln.  If  p,  p'  €  n„  are  two  different  solutions, 
the  inequalities  in  (3.4),  it  follows  that 

|p,-  -  p}|  <  26  (1  <  *  <  n) 

so  that  from  (3.9)  it  follows  that 

||p-  p'l!  <  26 

We  have  thus  proved 

Theorem  2. 


(3.6) 

(3.7) 

(3.8) 

(3.9) 
satisfying 

(3.10) 

(3.11) 


Let  us  assume  that 


(1)  e,  6,  8  satisfy  one  of  the  conditions  A,  B; 

(2)  the  arbitrary  sequence  C$  €  is  generated  by  an  independent  identically 


distributed  sequence  of  trials,  with  unknown  probabilities  pt-  (1  <  »  <  n). 


Then 


(a)  The  relation  (3.4)  holds. 

(b)  If  p,  p'  are  two  different  solutions,  their  distance  in  FIn  is  less  than  26. 


Remark  4. 


Let  L ^  be  the  Banach  space  of  all  vectors  (3.5)  with  norm  the  total  variation 


=  E  kl 


(3.12) 


Then  IIn  is  a  metric  space  with  distance 


IIIp -P'111  =  £  Ip,- rf I 


(3.13) 


where  p,  p'  €  nn. 

If  p,  p'  €  n„  are  two  different  solutions,  satisfying  (3.4),  it  follows  from  (3.13) 


IIIp  -  P'111  <  2 nS 


(3.14) 


It  is  easy  to  see  that 


Up  -  P'11  <  IIIp  -  P'111  <  »IIp-p'II 


(3.15) 


We  remark  also  that  if  L "  is  the  Euclidean  space  of  all  vectors  (3.5)  with 


norm 


From  (3.4)  it  follows  that 


VKj  0.298  <  p,  <  0.302;  0.698  <  p2  <  0.702  J  >  0.875 

and  from  (3.11)  we  obtain 

||p-  p'j|  <  0.004 

4.2.  Examples  under  Condition  B 

Example  3. 

Let  C%  be  a  sequence  with  n  =  2,  e  =  2^  =  0.125,  s  =  104,  6  > 
=  3  X  103,  m5>  =  7  X  103,  so  that 


—  1  -  —  =  0.46875 
2  2 


and  relation  (2.39)  takes  the  form 


dx  >  0.46875 


which  holds  for 


«  (e)  >  1.8 

Considering  Condition  B  in  form  (1.42)  it  is  easy  to  see  that  it  holds. 
(3.4)  it  follows  that 

wf  0.291  <  pj  <  0.309  ;  0.691  <  p2  <  0.709  )  >  0.875 


(4.3) 


(4.4) 


0.009, 


(4.5) 


(4.6) 


(4.7) 

From 


(4.8) 


and  from  (3.11)  it  follows  that 


||p- P'11  <  o.ois 


Example  4. 


Let  Cg  be  a  sequence  with  n  =  2,  e  =  Q-3  —  0.125,  s  —  106,  6  >  0.0009, 


m{  =  3  X  10s,  m2  =  7  X  10s;  in  this  case,  relations  (4.5)-(4.8)  hold,  so  that 


Condition  B  holds.  From  (3.4)  it  follows  that 


wj  0.2991  <  ?!  <  0.3009  ;  0.6991  <  p2  <  0.7009  J  >  0.875  (4.10) 


and  from  (3.11)  it  follows  that 


||p-  P'11  <  0.0018 


(4.11) 


4.3.  Examples  involving  images  that  satisfy  Condition  A  or  B 


Let  us  consider  a  digital  television  picture,  i.e.,  an  array  of  5002  points, 


where  each  point  can  have  256  levels  of  gray. 


Here  n  =  256,  s  =  5002  =  250,000;  let  e  =  — =  0.00390625. 

256 


Taking  these  values,  if  we  want  Condition  A  satisfied  it  is  sufficient  that 


4<52  X  250,000  X  — —  >  256 
256 


(4.12) 


106  &  >  2562  , 


(4.13) 


6  >  0.256 


(4.14) 


.  *: w  . 


*•  A.***  (v* -  ** -  *  ‘  **•  •*  .  k  ’  •  *.  •V“'»  '  *  *  -  *  •  ’  *  ■** 


Consequently 


with 


W 


<  0.256  ; 


(1  <  i  <  256)  >  0.9960937 


||p  -  tf\\  =  max  |  |p,  -  p\  \  ,  1  <  i  <  256  <  0.512 


Tiple  6. 

With  the  same  basic  data  as  in  Example  5,  we  take  n  =  256,  s  = 
£  ~  ^6  =  ®*00390625,  and  we  consider  that  Condition  B  holds,  i.e., 

2 6\fs  >  u  (f) 

Here 


~  ?  | ' 1  -  eokl  ~  1  <4  -  °-16667>  -  7  x  0  83334 

—  0.41667 


so  that  from  tables  it  follows  that 


Thus 


u  (e)  ~  1.30 


i.e.. 


26  X  500  >  1.30 


6  >  0.0013 


w\  —  -  Pi  <  0.0013  ;  (1  <  t  <  256)  j  >  0.9960937  (4.22) 


||p-  P'11  <  0.0026 


(4.23) 


Let  us  take  n  —  256,  s  —  5002,  e  =  — —  =  0.0625,  and  let  us  assume  that 

16 


Condition  A  holds.  Then 


so  that 


4^  X  250,000  X  —  >  256 

16 

(4.24) 

106  S2  >  212 

(4.25) 

6  >  0.064 

(4.26) 

Pi  <  0.064  ;  1  <  »'  <  256 }  >  0.9375 

(4.27) 

Up -P'11  <  0.128 

(4.28) 

Let  n  —  256,  s  —  250,000,  e  =  — =  0.0625  and  let  us  assume  that  Condi- 

16 


tion  B  holds.  Then 


V  >.*"  o'*  ■.‘'.’v'!  '.'‘o'  ^  -.'  v'  •-*.  • «-*  v  \ ' 


6 


=  (1  -  0.00025) 


j  X  0.99975  =  0.49987 


so  that 


u  (e)  ~  3.8 


(4.30) 


26  X  500  >  3.8 


(4.31) 


Thus 


6  >  0.0038 


(4.32) 


'{ - p,  <  0.0038  ;  1  <  i  <  256  j  >  0.9375 


||p-  P'11  <  0.0076 


(4.33) 


(4.34) 


nple  9. 


Let  us  assume  that  we  have  a  30-minute  sequence  of  TV  pictures.  If  we  have 
32  pictures  in  each  second,  we  have  a  total  of 


32  X  60  X  30  =  24  X  602 


(4.35) 


pictures,  succeeding  each  other  in  time.  Assuming  independence  between 
the  pictures,  we  have  n  —  258,  s  =  5002  X  24  X  602,  and  let 

e  =  =  0.00390625.  Assuming  that  Condition  A  holds,  the  value  of  6 

256 

is  given  by 


4^  (250,000)  X  24  X  602  X  >  256 


(4.36) 


106  a2  X  24  X  602  >  2562 


(4.37) 


Then 


103  6  X  22  X  60  >  256 


6  >  — -  >  0.001 


102  X  240 


(4.38) 


(4.39) 


Consequently 


'{  —  -  p,  <  0.001  ;  (1  <  i  <  256) }  >  0.9960937  (4.40) 


||p-  p^l  <  0.002 


(4.41) 


Let  us  consider  the  same  problem  as  in  Example  9,  with  the  supposition  that 
Condition  B  holds. 

In  this  case 


2 6  (500  X  22  X  60)  >  1.30 


(4.42) 


6  >  - — - ~  0.0000054 

2,400,000 


(4.43) 


so  that 


f  m?  \ 

Wj  —  -  Pi  <  0.0000054  ;  (1  <  i  <  256)  j  >  0.9960937 


||p  -  j/||  <  0.0000108 


(4.45) 
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